feat(media): add sharded media object-key paths by bradseiler · Pull Request #4533 · block/buzz

bradseiler · 2026-08-03T14:25:59Z

Why

Media payload keys are currently flat, so high-volume traffic lacks hash-leading S3 prefix entropy. Existing deployments also need a safe, explicit rollout that preserves old reads and rollback behavior without changing storage behavior merely by upgrading.

What

moves media payloads to media/<sha[0:2]>/<sha[2:4]>/<community>/<filename> while leaving Git/CAS namespaces unchanged
adds BUZZ_MEDIA_MIGRATION_PHASE with three complete media object-key path policies:
- legacy-only (default): read and write only legacy paths, preserving existing deployment behavior
- dual-read-and-write: prefer sharded reads with legacy fallback and write both paths
- sharded-only: read and write only sharded paths
defines the existing-deployment rollout as legacy-only → dual-read-and-write → backfill → sharded-only
recommends that new deployments start with sharded-only before their first upload, avoiding any future backfill or legacy cleanup
adds sharded-first/legacy-fallback read telemetry, strict bucket classification, logical billing deduplication, duplicate-layout gauges, and authoritative payload keys in upload records
includes buzz-media-layout-backfill and buzz-media-layout-delete-legacy in both relay image targets, with bounded paging, checkpoints, request-rate limiting, idempotency, destination verification, dry-run, and destructive confirmation
documents new-install and existing-deployment flows and provides Kubernetes Job examples

Validation

cargo test -p buzz-media --all-targets: 119 passed; live-MinIO test ignored
cargo test -p buzz-relay --lib config::tests::: 32 passed
cargo test -p buzz-relay --lib storage_sweep: 15 passed
cargo clippy -p buzz-media -p buzz-relay --all-targets -- -D warnings: clean
cargo fmt --all -- --check, desktop Tauri fmt, and git diff --check: clean
pre-push hooks after merging current origin/main: Rust, desktop, Tauri, mobile, and typecheck suites passed
Helm values/test YAML and values schema JSON parse cleanly; Helm CLI is unavailable locally

Generated with Goose

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>

Classify sharded payload keys in storage sweeps, preserve physical totals, and deduplicate logical legacy/sharded copies. Export read resolution, fallback, and duplicate-layout metrics. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>

Keep self-hosted upgrades on legacy writes by default, expose the write-layout gate through Helm and Compose, and document the explicit legacy-to-dual-to-sharded migration sequence. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>

Consolidate read and write policy behind one upgrade-safe phase and ship guarded maintenance binaries for backfill and legacy cleanup. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>

Signed-off-by: Brad Seiler <seiler@squareup.com>

Replace the Option + expect("checked above") pattern in both upload short-circuit paths with a filtered if-let binding, so the presence of the blob key is guaranteed structurally instead of by a comment on a non-local invariant. Also collapse the longhand match on existing_write_key into `?`. No behavior change. Signed-off-by: Brad Seiler <seiler@squareup.com>

Document that returning to legacy-only after accepting uploads in sharded-only makes sharded-layout objects unreadable until the phase is raised again. Ship sharded-only in the Compose example environment so fresh stacks never need a backfill, and correct the chart README to reflect that only the chart default remains legacy-only for upgrade safety. Signed-off-by: Brad Seiler <seiler@squareup.com>

Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>

tlongwell-block

Requesting changes for two brownfield blockers at head 1cfde2831e3b6512f24630b37f9ef130624d2f4e:

Legacy cleanup cannot safely resume from a sidecar checkpoint. verify_selected_destinations only builds verified_legacy_keys from sidecars at/after BUZZ_MEDIA_MIGRATION_START_AFTER, then deletes those shared flat CAS keys. One legacy key can serve multiple communities (as the function comment correctly notes). If community A sorts before the checkpoint and lacks its sharded copy while community B sorts after it and has a verified sharded copy, the resumed run verifies B, inserts the shared legacy key, and deletes it, stranding A. The README currently tells operators both tools can restart from any logged checkpoint. Either disallow start_after for destructive cleanup or make cleanup prove all sidecar bindings for every candidate legacy key across the full bucket before deletion. Please add the cross-community/resume regression test.
The stale branch revives the removed Helm media-auth flag without a merge conflict. A no-commit merge of this head into current origin/main cleanly restores relay.requireMediaGetAuth, BUZZ_REQUIRE_MEDIA_GET_AUTH, its schema entry, and render test. Current relay config explicitly classifies that variable as inert because media reads are unconditionally authenticated (crates/buzz-relay/src/config.rs:434-444,810-814). The resulting chart advertises a switch whose false value does nothing and emits a startup warning. Rebase and drop these unrelated stale additions.

What I traced:

Media payload read/head/range/stream and imeta checks route through migration candidates; default legacy-only preserves existing flat media reads/writes.
Sidecars and upload records retain their existing namespaces.
Git store files are untouched; Git keys remain packs/, manifests/, and repos/..., and the only shared S3 setting remains the pre-existing addressing style.

Verification at the exact PR head above:

PATH="$PWD/bin:$PATH" cargo test -p buzz-media --all-targets: 119 passed, live-MinIO test ignored.
PATH="$PWD/bin:$PATH" cargo test -p buzz-relay --all-targets: 879 passed across lib/bin, 40 ignored, 0 failed.
git diff --check: clean.
Current GitHub checks are green, but they do not exercise destructive cleanup resumption.

Ratings: Minimalness 6/10 (large but mostly justified migration surface; stale unrelated Helm flag is not), Elegance 8/10 (phase model and centralized key derivation are clear; cleanup checkpoint contract is misleading), Correctness 6/10 (upgrade default and live read paths look sound, but the destructive brownfield edge can cause data loss and the clean merge revives inert config). Not ready until both blockers are resolved.

This was referenced Aug 3, 2026

feat(media): add sharded compatibility reads #4015

Closed

feat(media): include payload keys in upload records #4016

Closed

feat(media): add configurable sharded writes #4017

Closed

bradseiler changed the title ~~feat(media): add phased S3 layout migration~~ feat(media): add sharded media object-key paths Aug 6, 2026

bradseiler marked this pull request as ready for review August 7, 2026 04:55

bradseiler requested a review from a team as a code owner August 7, 2026 04:55

bradseiler force-pushed the seiler/media-layout-migration branch from ada0dd3 to 828b51e Compare August 7, 2026 05:13

npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch and others added 16 commits August 10, 2026 16:12

fix(media): preserve legacy layout by default

a471969

Signed-off-by: Brad Seiler <seiler@squareup.com>

fix(media): simplify object-key path rollout

c897783

Signed-off-by: Brad Seiler <seiler@squareup.com>

fix(media): verify backfilled destination bytes

6ddc395

Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>

fix(media): verify legacy cleanup bytes

a821b3d

Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>

fix(media): read payloads through migration candidates

aa55a76

Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>

fix(media): verify cleanup destinations before deletion

1cfde28

Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>

bradseiler force-pushed the seiler/media-layout-migration branch from 5fc442c to 1cfde28 Compare August 10, 2026 20:17

tlongwell-block requested changes Aug 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(media): add sharded media object-key paths - #4533

feat(media): add sharded media object-key paths#4533
bradseiler wants to merge 16 commits into
mainfrom
seiler/media-layout-migration

bradseiler commented Aug 3, 2026 •

edited

Loading

Uh oh!

tlongwell-block left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bradseiler commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Validation

Uh oh!

tlongwell-block left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bradseiler commented Aug 3, 2026 •

edited

Loading